from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Mismatches between validation scores

WARNING! Mismatch between validation scores for 12 predictions (75.0%).
The observed differences can be found in the diff_accuracy_scores column.
The chosen difference threshold is 0.001 for accuracy_score.
See details in the dataframe below.

	estimator	function	diff_accuracy_scores	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup
1	KNeighborsClassifier_brute_force	predict	0.337	100000	1000	100	1.903170	0.139840	NaN	0.000420	0.001903	brute	-1	1	0.663	0.329361	0.011636	1.000	5.778371	5.781976
2	KNeighborsClassifier_brute_force	predict	0.243	100000	1	100	0.021827	0.002062	NaN	0.000037	0.021827	brute	-1	1	1.000	17.513739	0.118492	0.757	0.001246	0.001246
4	KNeighborsClassifier_brute_force	predict	0.125	100000	1000	100	2.720306	0.039169	NaN	0.000294	0.002720	brute	-1	5	0.757	17.876003	0.292534	0.882	0.152176	0.152197
7	KNeighborsClassifier_brute_force	predict	0.118	100000	1000	100	2.124504	0.052952	NaN	0.000377	0.002125	brute	1	100	0.882	0.359283	0.006211	1.000	5.913178	5.914062
8	KNeighborsClassifier_brute_force	predict	0.243	100000	1	100	0.019847	0.000414	NaN	0.000040	0.019847	brute	1	100	1.000	19.728844	0.212854	0.757	0.001006	0.001006
10	KNeighborsClassifier_brute_force	predict	0.219	100000	1000	100	3.008091	0.148458	NaN	0.000266	0.003008	brute	-1	100	0.882	19.131317	0.626016	0.663	0.157234	0.157318
13	KNeighborsClassifier_brute_force	predict	0.243	100000	1000	100	2.181953	0.055554	NaN	0.000367	0.002182	brute	1	5	0.757	0.264718	0.008835	1.000	8.242570	8.247159
14	KNeighborsClassifier_brute_force	predict	0.078	100000	1	100	0.020382	0.000513	NaN	0.000039	0.020382	brute	1	5	1.000	3.998132	0.052535	0.922	0.005098	0.005098
16	KNeighborsClassifier_brute_force	predict	0.266	100000	1000	100	1.151653	0.005095	NaN	0.000695	0.001152	brute	1	1	0.663	3.989154	0.079693	0.929	0.288696	0.288754
19	KNeighborsClassifier_brute_force	predict	0.104	100000	1000	2	1.674027	0.026270	NaN	0.000010	0.001674	brute	-1	1	0.896	0.278892	0.008957	1.000	6.002417	6.005512
20	KNeighborsClassifier_brute_force	predict	0.078	100000	1	2	0.004791	0.001588	NaN	0.000003	0.004791	brute	-1	1	1.000	3.974397	0.043678	0.922	0.001206	0.001206
22	KNeighborsClassifier_brute_force	predict	0.026	100000	1000	2	2.699315	0.041566	NaN	0.000006	0.002699	brute	-1	5	0.922	4.086712	0.080102	0.896	0.660510	0.660637

Raw results

fit

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	6.983	0.0	-1	1	19.324	0.702	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.001	7.156	0.0	-1	5	0.336	0.016	1.000	0.033	0.033	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.909	0.0	1	100	19.068	0.118	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.136	0.0	-1	100	0.344	0.012	1.000	0.033	0.033	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.681	0.0	1	5	3.924	0.133	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	6.962	0.0	1	1	0.263	0.011	1.000	0.044	0.044	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.334	0.0	-1	1	4.052	0.138	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.333	0.0	-1	5	0.266	0.006	1.000	0.018	0.018	See	See

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.903	0.140	0.000	0.002	-1	1	0.329	0.012	1.000	5.778	5.782	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.002	0.000	0.022	-1	1	17.514	0.118	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.720	0.039	0.000	0.003	-1	5	17.876	0.293	0.882	0.152	0.152	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.004	0.000	0.026	-1	5	0.344	0.012	1.000	0.075	0.075	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.125	0.053	0.000	0.002	1	100	0.359	0.006	1.000	5.913	5.914	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.000	0.000	0.020	1	100	19.729	0.213	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.008	0.148	0.000	0.003	-1	100	19.131	0.626	0.663	0.157	0.157	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	0.000	0.024	-1	100	0.336	0.010	1.000	0.073	0.073	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.182	0.056	0.000	0.002	1	5	0.265	0.009	1.000	8.243	8.247	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.001	0.000	0.020	1	5	3.998	0.053	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.152	0.005	0.001	0.001	1	1	3.989	0.080	0.929	0.289	0.289	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.002	0.000	0.020	1	1	0.260	0.007	1.000	0.076	0.076	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.674	0.026	0.000	0.002	-1	1	0.279	0.009	1.000	6.002	6.006	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.002	0.000	0.005	-1	1	3.974	0.044	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.699	0.042	0.000	0.003	-1	5	4.087	0.080	0.896	0.661	0.661	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.002	0.000	0.007	-1	5	0.275	0.010	1.000	0.026	0.026	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Mismatches between validation scores

WARNING! Mismatch between validation scores for 12 predictions (75.0%).
The observed differences can be found in the diff_accuracy_scores column.
The chosen difference threshold is 0.001 for accuracy_score.
See details in the dataframe below.

	estimator	function	diff_accuracy_scores	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup
1	KNeighborsClassifier_kd_tree	predict	0.071	1000000	1000	10	0.830243	0.991239	NaN	0.000096	0.000830	kd_tree	-1	1	0.929	2.941376	0.277718	1.000	0.282263	0.283519
2	KNeighborsClassifier_kd_tree	predict	0.054	1000000	1	10	0.002777	0.000501	NaN	0.000029	0.002777	kd_tree	-1	1	1.000	118.020665	0.000000	0.946	0.000024	0.000024
4	KNeighborsClassifier_kd_tree	predict	0.005	1000000	1000	10	0.976204	0.330636	NaN	0.000082	0.000976	kd_tree	-1	5	0.946	119.313901	0.000000	0.951	0.008182	0.008182
7	KNeighborsClassifier_kd_tree	predict	0.049	1000000	1000	10	5.363226	0.571489	NaN	0.000015	0.005363	kd_tree	1	100	0.951	2.929295	0.228344	1.000	1.830893	1.836448
8	KNeighborsClassifier_kd_tree	predict	0.054	1000000	1	10	0.003474	0.000831	NaN	0.000023	0.003474	kd_tree	1	100	1.000	117.339053	0.000000	0.946	0.000030	0.000030
10	KNeighborsClassifier_kd_tree	predict	0.022	1000000	1000	10	3.068582	0.125361	NaN	0.000026	0.003069	kd_tree	-1	100	0.951	118.978097	0.000000	0.929	0.025791	0.025791
13	KNeighborsClassifier_kd_tree	predict	0.054	1000000	1000	10	1.530345	0.213749	NaN	0.000052	0.001530	kd_tree	1	5	0.946	0.006334	0.000197	1.000	241.613077	241.729660
14	KNeighborsClassifier_kd_tree	predict	0.089	1000000	1	10	0.001671	0.000669	NaN	0.000048	0.001671	kd_tree	1	5	1.000	0.049015	0.011067	0.911	0.034091	0.034949
16	KNeighborsClassifier_kd_tree	predict	0.035	1000000	1000	10	0.855822	0.118249	NaN	0.000093	0.000856	kd_tree	1	1	0.929	0.066850	0.001973	0.894	12.802027	12.807599
19	KNeighborsClassifier_kd_tree	predict	0.109	1000	1000	2	0.028518	0.012147	NaN	0.000561	0.000029	kd_tree	-1	1	0.891	0.005730	0.000256	1.000	4.976586	4.981563
20	KNeighborsClassifier_kd_tree	predict	0.089	1000	1	2	0.002477	0.000202	NaN	0.000006	0.002477	kd_tree	-1	1	1.000	0.041165	0.002002	0.911	0.060176	0.060247
22	KNeighborsClassifier_kd_tree	predict	0.020	1000	1000	2	0.024493	0.001280	NaN	0.000653	0.000024	kd_tree	-1	5	0.911	0.040879	0.000684	0.891	0.599155	0.599238

Raw results

fit

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.926	0.089	0.027	0.0	-1	1	119.849	0.000	0.929	0.024	0.024	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.686	0.145	0.022	0.0	-1	5	2.872	0.214	1.000	1.283	1.287	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.588	0.141	0.022	0.0	1	100	120.896	0.000	0.951	0.030	0.030	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.674	0.087	0.022	0.0	-1	100	2.847	0.191	1.000	1.290	1.293	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.682	0.226	0.022	0.0	1	5	0.050	0.017	0.891	73.175	77.082	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.552	0.054	0.023	0.0	1	1	0.006	0.000	1.000	620.966	621.211	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.020	0.0	-1	1	0.064	0.003	0.894	0.012	0.012	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.024	0.0	-1	5	0.005	0.000	1.000	0.124	0.125	See	See

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.830	0.991	0.000	0.001	-1	1	2.941	0.278	1.000	0.282	0.284	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	118.021	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.976	0.331	0.000	0.001	-1	5	119.314	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.836	0.177	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.363	0.571	0.000	0.005	1	100	2.929	0.228	1.000	1.831	1.836	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	117.339	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.069	0.125	0.000	0.003	-1	100	118.978	0.000	0.929	0.026	0.026	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.985	0.300	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.530	0.214	0.000	0.002	1	5	0.006	0.000	1.000	241.613	241.730	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.001	0.000	0.002	1	5	0.049	0.011	0.911	0.034	0.035	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.856	0.118	0.000	0.001	1	1	0.067	0.002	0.894	12.802	12.808	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.006	0.000	1.000	0.186	0.186	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.029	0.012	0.001	0.000	-1	1	0.006	0.000	1.000	4.977	4.982	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.041	0.002	0.911	0.060	0.060	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.001	0.001	0.000	-1	5	0.041	0.001	0.891	0.599	0.599	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.005	0.000	1.000	0.425	0.425	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Mismatches between validation scores

WARNING! Mismatch between validation scores for 1 prediction (100.0%).
The observed differences can be found in the diff_accuracy_scores column.
The chosen difference threshold is 0.001 for accuracy_score.
See details in the dataframe below.

	estimator	function	diff_accuracy_scores	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	...	max_iter	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup
1	HistGradientBoostingClassifier_best	predict	0.176	100000	1000	100	0.124199	0.001895	300	0.006441	...	300	100	100	10	0.824	0.460697	0.008585	1.0	0.269589	0.269636

1 rows × 23 columns

Raw results

fit

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	108.663	0.0	300	0.001	0.001	0.561	0.028	0.824	193.741	193.985	See	See

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.124	0.002	300	0.006	0.0	0.461	0.009	1.0	0.27	0.27	See	See